This document relates to optimizing memory sharing in a virtualized computer system having multiple virtual machines wherein address space layout randomization (ASLR) or a similar technique is enabled in guest operating systems.
A virtual machine is a software-based abstraction of a physical computer system. In general, a computer program that can be executed on a physical computer system can be executed in a virtual machine using virtualization software. Virtualization software is software that is logically interposed and interfaces with a virtual machine and a physical computer system. Each virtual machine is configured to execute an operating system, referred to herein as a guest OS, and applications. A physical computer system, referred to herein as a host machine, can execute one or more virtual machines.
Virtualization software can allow two or more virtual machines to share memory pages. For example, a first virtual machine may use a system page having identical content as a system page used by a second virtual machine. The virtualization software can reduce the amount of memory used by the virtual machines by determining that the two virtual memory pages include identical content and only storing a single copy of the memory page in hardware memory, i.e., rather than storing an individual copy of the memory page for each virtual machine.
Some guest operating systems implement a computer security technique to protect from buffer overflow attacks. The security technique randomly arranges the positions of key data areas if a program, including the base of the executable and the positions of the stack, heap, and libraries in a process's address space, in order to prevent an attacker from reliably jumping to a particular exploited function in memory. However, using ASLR and similar techniques can cause the content of two memory pages that would otherwise be identical to be different. Thus, the number of memory pages that can be shared by virtualization software may be reduced as a result of using ASLR.
For example, certain Windows operating systems (such as Windows Vista) have a security feature referred to as ASLR, which is enabled by default. When this security feature is enabled, certain system drivers (.sys) and dynamic load libraries (.dll) are loaded to randomized base address. Thus, a particular .sys or .dll file has different base addresses in two virtual machines each running a copy of the Windows Vista operating system. As a result, the code section of a specific module, which makes calls to the particular .sys or .dll file, has different content, when stored on these two virtual machines. Therefore, the memory page storing the code section of the specific module thus cannot be shared between the virtual machines.